Adding Shield Tables #2315

dbermuehler · 2024-09-26T11:04:45Z

This PR adds all relevant Shield APIs as tables to Steampipe. It will add the following new tables:

aws_shield_protection
aws_shield_protection_group
aws_shield_attack
aws_shield_attack_summary
aws_shield_attack_statistic
aws_shield_subscription
aws_shield_drt_access
aws_shield_emergency_contact

Since this is my first time contributing to this project and also the first time I worked with go, I would be more than happy to receive some feedback especially on the following points:

Do you spot any obvious bugs?
Is the code utilising caching, pointers, etc. efficiently?
Is the way how I implemented the KeyColumns for the Get and List functions correct and can help to speed up the SQL queries?
Do you agree with the division of the different APIs into the aforementioned tables?
Do you agree with the names of the tables?
Do you agree on the way how I split up the API JSON response in the different columns, especially in the table aws_shield_emergency_contact and aws_shield_attack_statistic
Does the aws_shield_subscription table work properly if it is used with an AWS account that is not subscribed to Shield Advanced already?
How can I improve the code to better follow yours or go's best practices?

Looking forward to your feedback! Cheers!

Integration test logs

Logs

Add passing integration test logs here

Example query results

Results

`aws_shield_attack`

select
  resource_arn,
  start_time,
  end_time
from
  aws_shield_attack
where
  start_time between current_date - interval '30 day' and current_date;

+--------------------------------------------------------------+---------------------------+---------------------------+
| resource_arn                                                 | start_time                | end_time                  |
+--------------------------------------------------------------+---------------------------+---------------------------+
| arn:aws:cloudfront::XXXXXXXXXXXXX:distribution/YYYYYYYYYYYYY | 2024-10-22T09:10:00+02:00 | 2024-10-22T09:20:00+02:00 |
| arn:aws:cloudfront::XXXXXXXXXXXXX:distribution/YYYYYYYYYYYYY | 2024-10-20T17:09:00+02:00 | 2024-10-20T17:18:00+02:00 |
| arn:aws:cloudfront::XXXXXXXXXXXXX:distribution/YYYYYYYYYYYYY | 2024-10-06T10:33:00+02:00 | 2024-10-06T10:39:00+02:00 |
+--------------------------------------------------------------+---------------------------+---------------------------+

`aws_shield_attack_statistic`

select
  max,
  unit,
  attack_count
from
  aws_shield_attack_statistic
order by
  attack_count desc;

+--------------------+-------------------+--------------+
| max                | unit              | attack_count |
+--------------------+-------------------+--------------+
| 6439917.8166666668 | RequestsPerSecond | 38           |
+--------------------+-------------------+--------------+

`aws_shield_drt_access`

select
  role_arn,
  log_bucket_list
from
  aws_shield_drt_access;

+-------------------------------------------+-----------------+
| role_arn                                  | log_bucket_list |
+-------------------------------------------+-----------------+
| arn:aws:iam::XXXXXXXXXXXX:role/DRT-Role   | <null>          |
+-------------------------------------------+-----------------+

`aws_shield_emergency_contact`

select
  email_address,
  phone_number,
  contact_notes
from
  aws_shield_emergency_contact;

+------------------------------------+--------------+---------------------------------------------------------------------------------------------------------------------+
| email_address                      | phone_number | contact_notes                                                                                                       |
+------------------------------------+--------------+---------------------------------------------------------------------------------------------------------------------+
| [email protected] | <null>       | The email address provided notifies a member of the DDoS response team of company.com who is currently on-call. |
+------------------------------------+--------------+---------------------------------------------------------------------------------------------------------------------+

`aws_shield_protection`

select
  name,
  resource_arn
from
  aws_shield_protection;

+-------------------+--------------------------------------------------------------+
| name              | resource_arn                                                 |
+-------------------+--------------------------------------------------------------+
| company.com       | arn:aws:route53:::hostedzone/XXXXXXXXXXXXXX                  |
| distribution-1    | arn:aws:cloudfront::FFFFFFFFFFFF:distribution/YYYYYYYYYYYYYY |
| distribution-2    | arn:aws:cloudfront::FFFFFFFFFFFF:distribution/XXXXXXXXXXXXXX |
| distribution-3    | arn:aws:cloudfront::FFFFFFFFFFFF:distribution/ZZZZZZZZZZZZZZ |
| distribution-4    | arn:aws:cloudfront::FFFFFFFFFFFF:distribution/AAAAAAAAAAAAAA |
| distribution-5    | arn:aws:cloudfront::FFFFFFFFFFFF:distribution/BBBBBBBBBBBBBB |
| distribution-6    | arn:aws:cloudfront::FFFFFFFFFFFF:distribution/CCCCCCCCCCCCCC |
| distribution-7    | arn:aws:cloudfront::FFFFFFFFFFFF:distribution/EEEEEEEEEEEEEE |
+-------------------+--------------------------------------------------------------+

`aws_shield_subscription`

select
  subscription_state,
  start_time,
  end_time,
  auto_renew,
  proactive_engagement_status
from
  aws_shield_subscription;

+--------------------+---------------------------+---------------------------+------------+-----------------------------+
| subscription_state | start_time                | end_time                  | auto_renew | proactive_engagement_status |
+--------------------+---------------------------+---------------------------+------------+-----------------------------+
| ACTIVE             | 2023-01-01T14:59:53+02:00 | 2025-01-01T14:59:53+02:00 | ENABLED    | DISABLED                    |
+--------------------+---------------------------+---------------------------+------------+-----------------------------+

`aws_shield_protection_group`

select
  protection_group_id,
  aggregation,
  pattern,
  resource_type
from
  aws_shield_protection_group;

+---------------------+-------------+------------------+----------------------+
| protection_group_id | aggregation | pattern          | resource_type        |
+---------------------+-------------+------------------+----------------------+
| Test                | SUM         | BY_RESOURCE_TYPE | ROUTE_53_HOSTED_ZONE |
+---------------------+-------------+------------------+----------------------+

…e if List function is used

…g it to null

ParthaI

Hi @dbermuehler, I've provided a few initial review comments for the tables, particularly for aws_shield_attack. These comments can be applied across all the aws_shield_* tables.

Please take a look and make the necessary changes where applicable.

Additionally, the table documentation can be improved by adding more example queries by joining the tables.

Note: I see that we are updating the AWS SDK version to v1.31.0. Please ensure that this update does not affect any other tables.

Thanks!

aws/service.go

aws/plugin.go

aws/table_aws_shield_attack.go

aws/table_aws_shield_emergency_contact.go

Co-authored-by: ParthaI <[email protected]>

rogerioacp · 2024-10-23T11:45:44Z

Hey folks, this is an awesome PR! Our team will really benefit from having this tables in Steampipe.
Is there an expected deadline to merge it?

ParthaI

Hi @dbermuehler, I have left a few minor review comments. Please take look, thanks!

aws/table_aws_shield_attack.go

aws/table_aws_shield_protection.go

aws/table_aws_shield_protection_group.go

aws/table_aws_shield_protection.go

aws/table_aws_shield_protection_group.go

aws/table_aws_shield_protection.go

Co-authored-by: ParthaI <[email protected]>

ParthaI · 2024-10-25T04:26:17Z

@misraved, could you please review this PR when you have a chance and proceed if everything looks good?

Note: Regarding the optional qualifiers (start_time and end_time) for the aws_shield_attack table, we have decided to temporarily remove support for them. Handling the various combinations of start_time and end_time with different query operators is a bit complex at the moment.

You can find the related discussion in the community channel.

Your feedback would be greatly appreciated.

Thanks!

misraved · 2024-10-28T13:34:30Z

Thanks @dbermuehler for the new set of tables 👍!!

Could you please add query results(without any sensitive information) to the PR body?

dbermuehler · 2024-10-28T16:35:41Z

Thank you @misraved. I added the queries, including the (anonymized) query results.

dbermuehler added 2 commits September 26, 2024 12:02

Adding aws_shield_protection table

1ad40ee

Adding basic documentation

43f9053

dbermuehler marked this pull request as draft September 26, 2024 11:24

misraved added the hacktoberfest-accepted This pull request has been accepted for Hacktoberfest label Sep 26, 2024

misraved requested a review from ParthaI September 26, 2024 11:43

dbermuehler added 15 commits September 26, 2024 13:45

Creating Default-Region-Client

256a805

Fixed empty result bug

b07cd98

Adding resourceArn as key column for Get function

221cae5

Filtering for resource_type, resource_arn and protection name possibl…

4e5dd69

…e if List function is used

Adding DRT-Access, Emergency-Contact and Subscription table

ba6e4d6

Changed akas transform function

c64413b

Adding attack statistic table

3ea0895

Adding protection group table

767cc7e

Removing log statements

756bf4c

Adding tag support for the protection and protection group table

b104a45

Typo

5340c5b

Adding attack table

164fddd

Correcting key column

55ee9a6

Adding attack summary table

8dfefe6

Bug fixes

772faf4

dbermuehler changed the title ~~Adding aws_shield_protection table~~ Adding Shield Tables Sep 27, 2024

dbermuehler added 3 commits September 27, 2024 13:24

Adding comments to follow coding standard

1d4eaff

Adding subscription state to subscription table

68efc36

Adding documentation

294aeff

dbermuehler marked this pull request as ready for review September 27, 2024 14:46

dbermuehler added 2 commits September 30, 2024 07:23

Adding artificial priority column to emergency contact list

3e7e59c

If never enabled the ProactiveEngagementStatus can be empty -> mappin…

f10414c

…g it to null

ParthaI reviewed Oct 1, 2024

View reviewed changes

dbermuehler and others added 2 commits October 1, 2024 08:36

Apply suggestions from code review

a8c2b32

Co-authored-by: ParthaI <[email protected]>

Sorting tables in alphabetic order

b2f1a5b

dbermuehler added 12 commits October 1, 2024 08:36

Merge branch 'main' of github.com:dbermuehler/steampipe-plugin-aws

e18fabe

Adding error handling and remove redundant transform function

cc7b3a9

Changing column to subscription_state

0b6fbe4

Correcting Column Type

64a12f1

Merging aws_shield_attack_summary and aws_shield_attack tables into one

8be7416

Fixing small bug

283141f

Improving timerange code

92852b1

Downgraded Shield package to not upgrade aws-sdk-go-v2 version

2238a02

Addressing PR comments

2a29426

Fixed indentation

792d40d

Adding more queries to the documentation

d943d28

Removing start_time and end_time qualifier

fb973d4

ParthaI reviewed Oct 24, 2024

View reviewed changes

dbermuehler and others added 3 commits October 24, 2024 18:58

Apply suggestions from code review

8bd8673

Co-authored-by: ParthaI <[email protected]>

Added LIST FUNCTIONS comment

7f54f08

Added more SQL examples that contain joins

e5394dc

ParthaI approved these changes Oct 25, 2024

View reviewed changes

dbermuehler added 2 commits October 28, 2024 16:20

Fixing bug in documentation

1aa4c10

Fixing documentation

bc972ce

misraved approved these changes Oct 29, 2024

View reviewed changes

misraved merged commit d87d706 into turbot:main Oct 29, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding Shield Tables #2315

Adding Shield Tables #2315

dbermuehler commented Sep 26, 2024 •

edited

Loading

ParthaI left a comment

rogerioacp commented Oct 23, 2024

ParthaI left a comment

ParthaI commented Oct 25, 2024

misraved commented Oct 28, 2024

dbermuehler commented Oct 28, 2024 •

edited

Loading

Adding Shield Tables #2315

Adding Shield Tables #2315

Conversation

dbermuehler commented Sep 26, 2024 • edited Loading

Integration test logs

Example query results

aws_shield_attack

aws_shield_attack_statistic

aws_shield_drt_access

aws_shield_emergency_contact

aws_shield_protection

aws_shield_subscription

aws_shield_protection_group

ParthaI left a comment

Choose a reason for hiding this comment

rogerioacp commented Oct 23, 2024

ParthaI left a comment

Choose a reason for hiding this comment

ParthaI commented Oct 25, 2024

misraved commented Oct 28, 2024

dbermuehler commented Oct 28, 2024 • edited Loading

dbermuehler commented Sep 26, 2024 •

edited

Loading

`aws_shield_attack`

`aws_shield_attack_statistic`

`aws_shield_drt_access`

`aws_shield_emergency_contact`

`aws_shield_protection`

`aws_shield_subscription`

`aws_shield_protection_group`

dbermuehler commented Oct 28, 2024 •

edited

Loading